KNOWLEDGE DISTILLATION

Knowledge Distillation for Language Models (2025.naacl-tutorial)

Challenge:	Large language models are often inefficient for real-world deployment due to expensive inference costs.
Approach:	They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model.
Outcome:	The proposed method is the best for multi-lingual and multilingual student architectures.

Copied to clipboard

Yuqiao Wen, Freda Shi, Lili Mou

Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective (2023.findings-eacl)

Challenge:	Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation .
Approach:	They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation .
Outcome:	This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models.

Copied to clipboard

Jongwoo Ko, Seungjoon Park, Minchan Jeong, Sukjin Hong, Euijai Ahn, Du-Seong Chang, Se-Young Yun

An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition (2022.acl-long)

Challenge:	Existing methods for knowledge distillation (KD) are prone to overfitting to training datasets . recent advances in NLP have shown that using PLMs such as BERT and RoBERTa on downstream tasks is effective.
Approach:	They propose a consistency-regularized knowledge distillation method which mitigates overfitting of existing methods.
Outcome:	The proposed method outperforms existing methods on the GLUE benchmark and synthetic datasets.

Copied to clipboard

Zhuoran Li, Chunming Hu, Xiaohui Guo, Junfan Chen, Wenyi Qin, Richong Zhang

A Study of Non-autoregressive Model for Sequence Generation (2020.acl-main)

Challenge:	Existing models for named entity recognition only consider the potential transferability between two identical tasks across both domains.
Approach:	They propose to use a similarity metric model to improve cross-lingual named entity recognition task on target domain.
Outcome:	Empirical studies on 7 different languages confirm the effectiveness of the proposed model.

Copied to clipboard

Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, Tie-Yan Liu

Text Generation

Challenge:	Non-autoregressive (NAR) models generate all tokens in parallel, resulting in faster generation speed compared to autoregressive models.
Approach:	They propose to use knowledge distillation and source-target alignment to bridge the gap between NAR and autoregressive models in various tasks.
Outcome:	The proposed techniques can speed up NAR models in some tasks but not all . the proposed techniques reduce target token dependency while allowing for faster inference .

Natural Language Generation for Effective Knowledge Distillation (D19-61)

Copied to clipboard

Raphael Tang, Yao Lu, Jimmy Lin

Distilling Linguistic Context for Language Model Compression (2021.emnlp-main)

Challenge:	Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks.
Approach:	They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques .
Outcome:	The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability.

Copied to clipboard

Geondo Park, Gyeongman Kim, Eunho Yang

Dynamic Knowledge Distillation for Pre-trained Language Models (2021.emnlp-main)

Challenge:	Knowledge distillation is a major technique for deploying vast language models in resource-strapped environments.
Approach:	They propose a method that transfers contextual knowledge via Word Relation and Layer Transforming Relation.
Outcome:	The proposed method is able to transfer contextual knowledge without restrictions on architectural changes between teacher and student on language understanding tasks.

Copied to clipboard

Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun

Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition (2024.acl-srw)

Challenge:	Existing methods conduct knowledge distillation statically, e.g., student model aligns output distribution to teacher model on pre-defined training dataset.
Approach:	They propose a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency . they find it is promising and provide discussions on potential future directions towards more efficient methods .
Outcome:	The proposed method can boost student model performance while accelerating training . the proposed method reduces memory usage and accelerates model inference .

Copied to clipboard

Tyler McDonald, Ali Emami

Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor (2021.acl-long)

Challenge:	Trace-of-Thought Prompting allows small neural networks to emulate larger, teacher models with reduced computational demands.
Approach:	They propose a framework to distill critical reasoning capabilities from teacher models to student models . they use problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions .
Outcome:	a new framework enables small neural networks to emulate the performance of larger, teacher models . it leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions . the proposed framework is available on github.com/trace-of-thought/trac-of_thought-prompting/main .

Copied to clipboard

Xinyu Wang, Yong Jiang, Zhaohui Yan, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu

Green & Sustainable NLP

Challenge:	Knowledge distillation is a technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student).
Approach:	They propose a factorized form of the knowledge distillation objective for structured prediction which is tractable for many typical choices of the teacher and student models.
Outcome:	The proposed model is able to transfer knowledge between teacher and student models without loss of accuracy under four different scenarios.

Pretrained Bidirectional Distillation for Machine Translation (2023.acl-long)

Copied to clipboard

Yimeng Zhuang, Mei Tu

BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation (2025.coling-main)

Challenge:	Existing studies have focused on language knowledge transfer from pretrained models to neural machine translation models.
Approach:	They propose to use masked language pretraining to efficiently transfer bidirectional language knowledge to NMT models.
Outcome:	The proposed method can significantly improve machine translation performance and achieve competitive or even better results than previous methods.

Copied to clipboard

Minchong Li, Feng Zhou, Xiaohui Song

Using the Past Knowledge to Improve Sentiment Classification (2020.findings-emnlp)

Challenge:	Knowledge distillation (KD) is a method for reducing model size while preserving performance.
Approach:	They propose a method to distill large language models at the logit level by transferring knowledge from a large teacher model to a smaller student model.
Outcome:	The proposed method outperforms supervised fine-tuning, vanilla KL loss and five other distillation methods on 13 datasets.

Copied to clipboard

Qi Qin, Wenpeng Hu, Bing Liu

Information Retrieval

Challenge:	Existing model retains knowledge learned from past tasks and selectively transfers it to new task to help it learn better.
Approach:	They propose a lifelong learning model that can retain and selectively transfer the knowledge learned in the past to help learn the new task.
Outcome:	The proposed model outperforms strong baselines, including even multiple task learning.

Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)

Copied to clipboard

Joe Stacey, Marek Rei

Learning from Diverse Reasoning Paths with Routing and Collaboration (2025.emnlp-main)

Challenge:	Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits.
Approach:	They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target.
Outcome:	The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain.

Copied to clipboard

Zhenyu Lei, Zhen Tan, Song Wang, Yaochen Zhu, Zihan Chen, Yushun Dong, Jundong Li

Weight Distillation: Transferring the Knowledge in Neural Network Parameters (2021.acl-long)

Challenge:	Recent studies suggest that the reasoning abilities of large language models (LLMs) grows with model size and pre-training data.
Approach:	They propose to combine quality filtering, conditional routing, and cooperative peer teaching to transfer knowledge from powerful teacher models to compact and transparent students.
Outcome:	Experiments show that QR-Distill is superior to traditional methods.

Copied to clipboard

Ye Lin, Yanyang Li, Ziyang Wang, Bei Li, Quan Du, Tong Xiao, Jingbo Zhu

Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment (2025.coling-main)

Challenge:	Knowledge distillation is an effective method for model acceleration and compression.
Approach:	They propose to use parameters to distill knowledge from large neural networks to small ones . they propose to do this by using a parameter generator to transfer the knowledge to a small neural network .
Outcome:	The proposed method learns a small network 1.88 2.94x faster than the large network but with competitive BLEU points.

Copied to clipboard

Tianyu Peng, Jiajun Zhang

Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation (2022.acl-long)

Challenge:	Existing knowledge distillation techniques for large language models are causing difficulties for student models to learn multi-modal probability distributions.
Approach:	They propose a ranking loss-based knowledge distillation method that encourages consistency of the ranking of peak predictions between teacher and student models.
Outcome:	The proposed method improves student models' ability to learn multi-modal distributions.

Copied to clipboard

Liang Ding, Longyue Wang, Shuming Shi, Dacheng Tao, Zhaopeng Tu

Multilinguality

Challenge:	Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation models, but it can lose important information for translating low-frequency words.
Approach:	They propose a knowledge distillation method which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data.
Outcome:	Extensive experiments on eight WMT benchmarks show that monolingual KD outperforms the standard KD by improving low-frequency word translation without introducing any computational cost.

ThinkAnswer Loss: Balancing Semantic Similarity and Exact Matching for LLM Reasoning Enhancement (2025.findings-emnlp)

Copied to clipboard

Shan Yang, Kun Wu, Zeju Li, Linlin Zhang, Xiangyu Pei, Leike An, Yu Liu

Reasoning

Challenge:	Existing methods for knowledge distillation use Chain-of-Thought (CoT) and answer pairs, but they lack appropriate supervision signals.
Approach:	They propose a framework that decouples CoT and answer supervision . the framework applies semantic similarity constraints while maintaining strict literal matching for the answer .
Outcome:	The proposed framework decouples CoT and answer supervision while maintaining strict literal matching for the answer.

Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)

Copied to clipboard

Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, Juanzi Li

Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation (2024.findings-emnlp)

Challenge:	Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression.
Approach:	They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM .
Outcome:	The proposed model can be used to distill a 1.9B student model using a teacher LLM.

Copied to clipboard

Yuhang Zhou, Jing Zhu, Paiheng Xu, Xiaoyu Liu, Xiyao Wang, Danai Koutra, Wei Ai, Furong Huang

Multi-Granularity Semantic Revision for Large Language Model Distillation (2026.acl-long)

Challenge:	Knowledge distillation (KD) is a promising solution for large language models, but their deployment remains computationally expensive.
Approach:	They propose a framework which iteratively balances training data within a fixed computational budget and enables the transfer of knowledge from expensive teacher LLMs to smaller student models.
Outcome:	The proposed framework achieves state-of-the-art performance across diverse long-tailed datasets, enhancing both the efficiency and efficacy of the distilled models.

Copied to clipboard

Xiaoyu Liu, Yun Zhang, Wei Li, Simiao Li, Xudong Huang, Hanting Chen, Yehui Tang, Jie Hu, Zhiwei Xiong, Yunhe Wang

Annealing Knowledge Distillation (2021.eacl-main)

Challenge:	Existing methods for generating large language models rely on student-generated outputs, which introduce generation errors and misguide the distillation process.
Approach:	They propose a multi-granularity semantic revision method for LLM distillation that corrects errors using teacher-generated tokens and re-generates the sequence to minimize errors.
Outcome:	The proposed method reduces errors and misguides distillation on student models and improves consistency between teacher and student outputs.

Copied to clipboard

Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, Ali Ghodsi

Lifting the Curse of Capacity Gap in Distilling Language Models (2023.acl-long)

Challenge:	Knowledge distillation (KD) is a powerful model compression technique for deep neural networks.
Approach:	They propose a method to feed the rich information provided by teacher’s soft-targets incrementally and more efficiently by annealing the teacher output incrementally.
Outcome:	The proposed method can be used on image classification and NLP language inference tasks with BERT-based models on the GLUE benchmark.

Copied to clipboard

Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, Dawei Song

DTDES-KGE: Dual-Teacher Knowledge Distillation with Distinct Embedding Spaces for Knowledge Graph Embeddings (2025.findings-emnlp)

Challenge:	Existing studies have shown that pretrained language models require a tremendous amount of inference compute to perform.
Approach:	They propose to compress pretrained language models to small ones with a teacher-student paradigm to fill the capacity gap.
Outcome:	The proposed model achieves state-of-the-art performance at small FLOPs compared with competitive baselines.

Copied to clipboard

Bofan Wei, Hongyuan Xu, Yuhang Niu, Jiarui Ren, Yanlong Wen, Xiaojie Yuan

Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation (2021.acl-long)

Challenge:	Existing knowledge distillation methods rely on a single teacher embedding space . existing methods overlook valuable complementary knowledge from teachers in distinct embeddable spaces.
Approach:	They propose a knowledge distillation framework that leverages dual teachers in embedding spaces to enhance performance.
Outcome:	The proposed framework significantly improves knowledge distillation performance by leveraging dual teachers in distinct embedding spaces.

Copied to clipboard

Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, Zhaopeng Tu

Multilinguality

Challenge:	Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation models.
Approach:	They propose to use knowledge distillation to generate training data for non-autoregressive translation models by leveraging pretraining.
Outcome:	The proposed approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets.

GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation (2024.findings-naacl)

Copied to clipboard

Mohsen Gholami, Mohammad Akbari, Tianxi Hu, Vaden Masrani, Z. Wang, Yong Zhang

Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models (2026.findings-acl)

Challenge:	Existing methods to generate data using LLMs are limited by sampling from the center of original content distribution.
Approach:	They propose a task-agnostic data generation and knowledge distillation framework for LLMs that employs an iterative out-of-distribution-guided feedback mechanism to generate data.
Outcome:	The proposed framework outperforms prior arts and the LLM on 10 different classification tasks and noisey generated data.

Copied to clipboard

Mengxiang Zhang, Lingyuan Liu

MTA4DPR: Multi-Teaching-Assistants Based Iterative Knowledge Distillation for Dense Passage Retrieval (2024.emnlp-main)

Challenge:	Existing methods for knowledge distillation address the teacher–student capacity gap by mixing teacher and student distributions in the distillation target or using curriculum learning to sequence training from easy to hard examples.
Approach:	They propose a white-box KD framework that co-designs curriculum scheduling and target mixing through a unified difficulty-aware principle.
Outcome:	The proposed framework outperforms existing methods while reducing training runtime by over 10%.

Copied to clipboard

Qixi Lu, Endong Xun, Gongbo Tang

Information Retrieval

Challenge:	Existing studies have shown the effectiveness of knowledge distillation in DPR, but there is a performance gap between the teacher and the distilled student.
Approach:	They propose an iterative knowledge distillation method which transfers knowledge from teacher to student with help of multiple assistants in an iterated manner.
Outcome:	The proposed method achieves state-of-the-art performance among models with same parameters on multiple datasets and is competitive when compared with larger models.

EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments (2025.emnlp-main)

Copied to clipboard

Minh-Phuc Truong, Hai An Vu, Tu Vu, Nguyen Thi Ngoc Diep, Linh Ngo Van, Thien Huu Nguyen, Trung Le

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers (2021.findings-acl)

Challenge:	Existing methods for knowledge distillation focus on direct output alignment, neglecting this crucial structural information.
Approach:	They propose a framework for knowledge distillation that maps tokens one-to-one and aligns attention matrix patterns using Centered Kernel Alignment.
Outcome:	The proposed framework significantly outperforms existing CTKD baselines.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Yongfeng Huang

ToDi: Token-wise Distillation via Fine-Grained Divergence Control (2025.emnlp-main)

Challenge:	Pre-trained language models (PLMs) have huge model sizes and computational complexity, making it difficult to deploy them to low-latency and high-concurrence online systems.
Approach:	They propose a multi-teacher knowledge distillation framework for pre-trained language model compression that can train high-quality student model from multiple teacher PLMs.
Outcome:	The proposed framework can train high-quality student model from multiple teacher PLMs with shared pooling and prediction layers to align output space for better collaborative teaching.

Copied to clipboard

Seongryong Jung, Suwan Yoon, DongGeon Kim, Hwanhee Lee

Learning to Maximize Mutual Information for Chain-of-Thought Distillation (2024.findings-acl)

Challenge:	Large language models (LLMs) offer impressive performance but are impractical for resource-constrained deployment due to high latency and energy consumption.
Approach:	They propose a method that adaptively combines FKL and RKL per token using a sigmoid-based weighting function derived from the teacher-student probability log-ratio.
Outcome:	The proposed method outperforms baselines using uniform or less granular strategies across instruction-following benchmarks.

Copied to clipboard

Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, Ke Ding

CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation (2022.coling-1)

Challenge:	Knowledge distillation is a technique of transferring knowledge from large, complex models to smaller ones.
Approach:	They propose a method utilizing chain-of-thought distillation to transfer knowledge from large, complex models to smaller ones by maximizing mutual information of the representation features of the two tasks.
Outcome:	The proposed method outperforms the state-of-the-art knowledge distillation method on four datasets.

Copied to clipboard

Md Akmal Haidar, Mehdi Rezagholizadeh, Abbas Ghaddar, Khalil Bibi, Phillippe Langlais, Pascal Poupart

On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)

Challenge:	Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models.
Approach:	They propose a data augmentation technique tailored for knowledge distillation based on contrastive loss to improve masked adversarial data augmented by intermediate layer matching.
Outcome:	The proposed technique outperforms state-of-the-art methods on the GLUE benchmark and in an out-of domain evaluation.

Copied to clipboard

Tianyang Zhao, Kunwar Yashraj Singh, Srikar Appalaraju, Peng Tang, Ying Nian Wu, Li Erran Li

Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model Compression (2022.coling-1)

Challenge:	Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining.
Approach:	They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss.
Outcome:	The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures.

Copied to clipboard

Xinge Ma, Jin Wang, Liang-Chih Yu, Xuejie Zhang

Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation (2023.acl-long)

Challenge:	Knowledge distillation (KD) can transfer knowledge from the original model into a compact model to achieve model compression.
Approach:	They propose a knowledge distillation method with reptile meta-learning to facilitate the transfer of knowledge from the teacher to the student.
Outcome:	Extensive experiments on the GLUE benchmark show the proposed method performs better than previous methods.

Copied to clipboard

Songming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen, Wenjuan Han, Jian Liu, Jinan Xu

Understanding and Improving Knowledge Distillation for Quantization Aware Training of Large Transformer Encoders (2022.emnlp-main)

Challenge:	Existing knowledge distillation techniques for neural machine translation lack special treatment on the top-1 information, which is limiting the potential of KD.
Approach:	They propose a method to distill knowledge from top-1 predictions of teachers and a technique to infuse more additional knowledge by distilling on the data without ground-truth targets.
Outcome:	The proposed method outperforms the vanilla word-level KD and outperfies the existing methods on three different students with different capacity gaps.

Copied to clipboard

Minsoo Kim, Sihwa Lee, Suk-Jin Hong, Du-Seong Chang, Jungwook Choi

Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training (2025.findings-emnlp)

Challenge:	Knowledge distillation (KD) has been used for quantization-aware training to improve the ability of a lightweight model with the transferred knowledge from the teacher.
Approach:	They propose two methods to improve attention recovery of quantized large Transformers by combining attention-map and attention-output losses.
Outcome:	The proposed methods achieve state-of-the-art accuracy for quantized large Transformer encoder models with sub-2-bit weight quantization.

Copied to clipboard

Mengxiang Zhang, Lingyuan Liu

Cost-effective Distillation of Large Language Models (2023.findings-acl)

Challenge:	Knowledge distillation (KD) enables the compression of large language models (LLMs) conventional methods suffer from training-inference mismatches and suboptimal performance due to expensive student-generated outputs.
Approach:	They propose a method that combines a CL strategy and adaptive loss design to reduce training mismatches and suboptimal performance.
Outcome:	L2M-KD outperforms existing white-box KD methods on instruction-following tasks . it outperformed existing methods, achieving superior student model performance with reduced overhead .

Copied to clipboard

Sayantan Dasgupta, Trevor Cohn, Timothy Baldwin

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)

Challenge:	Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets.
Approach:	They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture.
Outcome:	The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets.

Copied to clipboard

Chenglong Wang, Yi Lu, Yongyu Mu, Yimin Hu, Tong Xiao, Jingbo Zhu

AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)

Challenge:	Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model.
Approach:	They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation.
Outcome:	The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets.

Copied to clipboard

Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Rui Wang

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting (2026.acl-long)

Challenge:	Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information.
Approach:	They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model.
Outcome:	The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods.

Copied to clipboard

Xinhang Ma, William Yeoh, Ning Zhang, Yevgeniy Vorobeychik

Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021.emnlp-main)

Challenge:	Knowledge distillation is a widely adopted technique for transferring capabilities from LLMs to smaller, more efficient student models.
Approach:	They propose methods for modifying teacher-generated reasoning traces to achieve two objectives that deter unauthorized distillation: (1) anti-distillation, or degrading the training usefulness of query responses; and (2) API watermarking, which embeds verifiable signatures in student models.
Outcome:	The proposed method achieves strong anti-distillation effect while maintaining or even improving teacher performance.

Copied to clipboard

Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, Mehdi Rezagholizadeh

Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)

Challenge:	Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions.
Approach:	They propose to use teacher training data for model compression . they investigate six tasks and find they can achieve between 75% and 92% of the teacher’s classification score while compressing the model 30 times.
Outcome:	The proposed solution achieves between 75% and 92% of the teacher’s classification score while compressing the model 30 times.

Copied to clipboard

Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, Dacheng Tao

f-Divergence Minimization for Sequence-Level Knowledge Distillation (2023.acl-long)

Challenge:	Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications.
Approach:	They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model.
Outcome:	The proposed method can achieve consistent and significant performance gains across all model types and sizes.

Copied to clipboard

Yuqiao Wen, Zichao Li, Wenyu Du, Lili Mou

One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)

Challenge:	Existing knowledge distillation approaches focus on minimizing a generalized f-divergence function.
Approach:	They propose a framework which formulates sequence-level knowledge distillation as minimizing a generalized f-divergence function.
Outcome:	The proposed framework outperforms existing methods and reduces intractable divergence to word-level losses.

Copied to clipboard

Xiaoqin Chang, Sophia Yat Mei Lee, Suyang Zhu, Shoushan Li, Guodong Zhou

Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models (2023.acl-long)

Challenge:	Existing knowledge distillation models require large computing resources and long inference time to perform.
Approach:	They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning.
Outcome:	The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks.

Copied to clipboard

Junmo Kang, Wei Xu, Alan Ritter

Self-Evolution Knowledge Distillation for LLM-based Machine Translation (2025.coling-main)

Challenge:	Increasing the size of pre-trained models can consistently improve performance on downstream tasks after fine-tuning, as seen in studies based on BERT, RoBERTa, T5 and empirical scaling laws.
Approach:	They propose to use knowledge distillation to build a compact model with a fixed budget instead of annotating data and manually labeling it.
Outcome:	The proposed approach reduces inference costs by reducing costs by hiring annotators and labelling data.

Copied to clipboard

Yuncheng Song, Liang Ding, Changtong Zan, Shujian Huang

Rethinking Task-Specific Knowledge Distillation: Contextualized Corpus as Better Textbook (2022.emnlp-main)

Challenge:	Existing knowledge distillation strategies for large language models minimize output distributions between student and teacher models indiscriminately for each token.
Approach:	They propose a distillation strategy that integrates teacher and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process.
Outcome:	The proposed method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets.

Copied to clipboard

Chang Liu, Chongyang Tao, Jianxin Liang, Tao Shen, Jiazhan Feng, Quzhe Huang, Dongyan Zhao

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities (2025.findings-acl)

Challenge:	Existing methods for knowledge distillation use a two-stage paradigm: general distillation with a task-agnostic general corpus and task-specific distillation using augmented task- specific corpus.
Approach:	They propose a contextualized corpus that contextualizes task corpus with large-scale general corpus through relevance-based text retrieval to improve student learning.
Outcome:	The proposed model improves on the GLUE benchmark and shows that it is better than generalized corpus and augmented task-specific corpus.

Copied to clipboard

Zhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo, Dahua Lin, Xiaoliang Wang, Nguyen Cam-Tu, Fei Tan

Dialogue Systems & Conversational Agents

Challenge:	Knowledge distillation (KD) with Direct Preference Optimization (DPO) has emerged as a promising approach to enhance the conversational abilities of smaller models using a larger teacher model.
Approach:	They propose a framework that integrates the teacher's distributional information into DPO distillation while preserving theoretical guarantees.
Outcome:	The proposed framework outperforms existing methods in restoring performance for pruned models and enhancing smaller models within the same LLM family.

Learn From One Specialized Sub-Teacher: One-to-One Mapping for Feature-Based Knowledge Distillation (2023.findings-emnlp)

Copied to clipboard

Khouloud Saadi, Jelena Mitrović, Michael Granitzer

Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs (2025.acl-long)

Challenge:	Empirical results show that our proposed approach outperforms the state-of-the-art methods by maintaining higher performance on most benchmark datasets.
Approach:	They propose to break down the global feature distillation task into N local sub-tasks and make each focused sub-student learn from one specialized sub-teacher.
Outcome:	The proposed method outperforms state-of-the-art methods on most benchmark datasets while maintaining higher performance.

Copied to clipboard

Anshumann Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee

Distillation Traps and Guards: A Calibration Knob for LLM Distillability (2026.acl-long)

Challenge:	Knowledge distillation is a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached.
Approach:	They propose an importance-sampling-based method which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits.
Outcome:	The proposed method enables faster training of student models with marginal overhead (10%) compared to cross-entropy based training, while maintaining competitive performance compared with full distillation.

Copied to clipboard

Weixiao Zhan, Yongcheng Jing, Leszek Rutkowski, Dacheng Tao

On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025.findings-acl)

Challenge:	Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks.
Approach:	They propose a method that allows teachers to control their distillability via reinforcement fine-tuning (RFT) they propose to use tail noise, off-policy instability, and the teacher–student gap to improve KD.
Outcome:	The proposed method outperforms SFT and KD baselines and can be used to protect teachers and students from bottlenecks.

Copied to clipboard

Suhas Kamasetty Ramesh, Ayan Sengupta, Tanmoy Chakraborty

PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024.findings-acl)

Challenge:	Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance.
Approach:	They propose to use knowledge distillation to compress large language models into smaller ones while preserving performance.
Outcome:	The proposed technique improves the performance of smaller models by 10% while providing only marginal benefits for larger models.

Copied to clipboard

Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang

Dual-Space Knowledge Distillation for Large Language Models (2024.emnlp-main)

Challenge:	Knowledge distillation (KD) is a technique for transferring expertise from large teacher models to compact student models with reduced memory footprints and inference costs.
Approach:	They propose to transfer knowledge from large teacher models to compact student models by exploiting teacher-student capacity discrepancies to generate pseudo-preference pairs where teacher outputs are preferred over student outputs.
Outcome:	The proposed framework exploits teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs.

Copied to clipboard

Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, Jinan Xu

Maximizing the Effectiveness of Larger BERT Models for Compression (2025.acl-long)

Challenge:	Existing large language models (LLMs) have strong generalization abilities due to their huge model capacities.
Approach:	They propose a dual-space knowledge distillation framework that unifies the output spaces of the two models for KD.
Outcome:	The proposed framework outperforms existing white-box KD frameworks on task-agnostic instruction-following benchmarks and can automatically align representations of two models with different vocabularies.

Copied to clipboard

Wen-Shu Fan, Su Lu, Shangyu Xing, Xin-Chun Li, De-Chuan Zhan

Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching (2025.acl-long)

Challenge:	Existing methods for capturing large BERT models as teachers do not fully exploit the potential advantages of larger teachers.
Approach:	They propose a method that leverages a pretrained teacher model to guide the training of a lightweight student model to enhance knowledge transfer.
Outcome:	The proposed method enhances knowledge transfer by leveraging a pretrained teacher model to guide the training of a lightweight student model.

Copied to clipboard

Mingzhe Li, Jing Xiang, Qishen Zhang, Kaiyang Wan, Xiuying Chen

Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation (2025.acl-long)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable capabilities in acquiring diverse knowledge, making them highly effective across a wide range of tasks.
Approach:	They propose a flipped knowledge distillation paradigm where LLM learns from SLM . they propose to reinterpret LLMs as encoder-decoder models using LoRA .
Outcome:	The proposed model has been deployed in an online application environment and validated on financial and healthcare benchmarks and real-world applications.

Copied to clipboard

Guoqiang Gong, Jiaxing Wang, Jin Xu, Deping Xiang, Zicheng Zhang, Leqi Shen, Yifeng Zhang, JunhuaShu JunhuaShu, ZhaolongXing ZhaolongXing, Zhen Chen, Pengzhang Liu, Ke Zhang

Sinkhorn Distance Minimization for Knowledge Distillation (2024.lrec-main)

Challenge:	Knowledge distillation (KD) compresses large language models into lightweight versions called student models.
Approach:	They propose to align the entire feature dynamics between teacher and student models by using two additional loss terms to achieve this.
Outcome:	The proposed method matches the entire feature dynamics between teacher and student models rather than just the final states.

Copied to clipboard

Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, Houqiang Li

MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation (2026.acl-long)

Challenge:	Existing knowledge distillation methods investigate divergence measures but fail to deliver effective supervision when few distribution overlap exists between teacher and student.
Approach:	They propose a knowledge distillation method that exploits the Sinkhorn distance to ensure a nuanced assessment of the disparity between teacher and student distributions.
Outcome:	The proposed method outperforms state-of-the-art methods on all kinds of LLMs with encoder-only, encoder decoder, and decoded architectures.

Copied to clipboard

Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen

Pru-CoT: Towards Efficient Reasoning Distillation via Pruning Chain-of-Thought (2026.findings-acl)

Challenge:	Existing methods for reducing the computational cost of large language models (LLMs) focus on minimizing the divergence between the output probability distributions of the teacher and the student, which limits knowledge transfer.
Approach:	They propose a framework that aligns teacher and student representations along their layer-wise transformation trajectory.
Outcome:	The proposed framework outperforms state-of-the-art benchmarks on teacher–student layers.

Copied to clipboard

Han Liu, Shuotian Ma, Hui Li, Xiaotong Zhang, Fenglong Ma, Hong Yu

Stable On-Policy Distillation through Adaptive Target Reformulation (2026.findings-acl)

Challenge:	Existing heuristics fail to capture global causal logic due to rigid rules and limited search spaces.
Approach:	They propose a framework that extracts the essential logical structure from reasoning chains.
Outcome:	Experiments show that Pru-CoT models generate more compact reasoning paths compared to models trained on verbose data.

Copied to clipboard

Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggyu Lim, Taesup Kim